Benchmark Design for Robust Profile-Directed Optimization

Profile-guided code transformations specialize program code according to the profile provided by execution on training data. Consequently, the performance of the code generated usind this feedback is sensitive to the selection of training data. Used in this fashion, the principle behind profileguided optimization techniques is the same as off-line learning commonly used in the field of machine learning. However, scant use of proper validation techniques for profileguided optimizations have appeared in the literature. Given the broad use of SPEC benchmarks in the computer architecture and optimizing compiler communities, SPEC is in a position to influence the proper evaluation and validation of profile-guided optimizations. Thus, we propose an evaluation methodology appropriate for profile-guided optimization based on cross-validation. Cross-validation is a methodology from machine learning that takes input sensitivity into account, and provides a measure of the generalizability of results.